Learning Resources

Textbooks and References

• “R for Data Science” by Hadley Wickham and Garrett Grolemund:
  • Free Online Book: https://r4ds.had.co.nz
• Visit bookdown site: https://bookdown.org
  • Many more on the archive page.

Interactive Exercises

• Posit Primers:https://posit.cloud/learn/primers:
  • The Basics, Work with Data, Visualize Data, Tidy Your Data, Report Reproducibly
• {swirl} Learn R, in R: https://swirlstats.com
  • Designed and developed by a team at Johns Hopkins University for coursera courses

Posit Primers created by learnr

• learnr Interactive Tutorials for R

Data Science and EDA

Wikipedia https://en.wikipedia.org/wiki/Data_science

An inter-disciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights from many structural and unstructured data.

• Create Insights
• Impact Decision Making
• Maintain & Improve Overtime

What is R?

R (programming language), Wikipedia

• R is a programming language and free software environment for statistical computing and graphics supported by the R Foundation for Statistical Computing.

• The R language is widely used among statisticians and data miners for developing statistical software and data analysis.

• A GNU package, the official R software environment is written primarily in C, Fortran, and R itself (thus, it is partially self-hosting) and is freely available under the GNU General Public License.

History of R and more

“R Programming for Data Science” by Roger Peng

• Chapter 2. History and Overview of R
• Overview and History of R: Youtube video

Why R? – Responses by Hadley Wickham

r4ds: R is a great place to start your data science journey because

• R is an environment designed from the ground up to support data science.
• R is not just a programming language, but it is also an interactive environment for doing data science.
• To support interaction, R is a much more flexible language than many of its peers.

Why R today?

When you talk about choosing programming languages, I always say you shouldn’t pick them based on technical merits, but rather pick them based on the community. And I think the R community is like really, really strong, vibrant, free, welcoming, and embraces a wide range of domains. So, if there are like people like you using R, then your life is going to be much easier. That’s the first reason.

Interview: “Advice to Young (and Old) Programmers, H. Wickham”

What is RStudio? https://posit.com

RStudio is an integrated development environment, or IDE, for R programming.

R Studio (Wikipedia)

RStudio is an integrated development environment (IDE) for R, a programming language for statistical computing and graphics. It is available in two formats: RStudio Desktop is a regular desktop application while RStudio Server runs on a remote server and allows accessing RStudio using a web browser.

Installation of R and R Studio

R Installation

To download R, go to CRAN, the comprehensive R archive network. CRAN is composed of a set of mirror servers distributed around the world and is used to distribute R and R packages. Don’t try and pick a mirror that’s close to you: instead use the cloud mirror, https://cloud.r-project.org, which automatically figures it out for you.

A new major version of R comes out once a year, and there are 2-3 minor releases each year. It’s a good idea to update regularly.

R Studio Installation

Download and install it from http://www.rstudio.com/download.

RStudio is updated a couple of times a year. When a new version is available, RStudio will let you know.

R Studio

The First Step

1. Start R Studio Application
2. Top Menu: File > New Project > New Directory > New Project > Directory name or Browse the directory and choose the parent directory you want to create the directory

When You Start the Project

1. Go to the directory you created
2. Double click _‘Directory Name’.Rproj

Or,

1. Start R Studio
2. File > Open Project (or choose from Recent Project)

In this way the working directory of the session is set to the project directory and R can search releted files without difficulty (getwd(), setwd())

Posit Cloud

RStudio Cloud is a lightweight, cloud-based solution that allows anyone to do, share, teach and learn data science online.

Cloud Free

• Up to 15 projects total
• 1 shared space (5 members and 10 projects max)
• 15 project hours per month
• Up to 1 GB RAM per project
• Up to 1 CPU per project
• Up to 1 hour background execution time

How to Start Posit Cloud

1. Go to https://posit.cloud/
2. Sign Up: top right

• Email address or Google account

1. New Project: Project Name
2. R Console

Start RStudio and create a project, or login to Posit Cloud and create a project.

The First Examples

Input the following codes into Console in the left bottom pane.

• The first two:

head(cars)

str(cars)

## 'data.frame':    50 obs. of  2 variables:
##  $ speed: num  4 4 7 7 8 9 10 10 10 11 ...
##  $ dist : num  2 10 4 22 16 10 18 26 34 17 ...

• Two more:

summary(cars)

##      speed           dist       
##  Min.   : 4.0   Min.   :  2.00  
##  1st Qu.:12.0   1st Qu.: 26.00  
##  Median :15.0   Median : 36.00  
##  Mean   :15.4   Mean   : 42.98  
##  3rd Qu.:19.0   3rd Qu.: 56.00  
##  Max.   :25.0   Max.   :120.00

plot(cars)

• And three more:

plot(cars) # cars: Speed and Stopping Distances of Cars
abline(lm(cars$dist~cars$speed))

lm(cars$dist~cars$speed)

## 
## Call:
## lm(formula = cars$dist ~ cars$speed)
## 
## Coefficients:
## (Intercept)   cars$speed  
##     -17.579        3.932

summary(lm(cars$dist~cars$speed))

## 
## Call:
## lm(formula = cars$dist ~ cars$speed)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -29.069  -9.525  -2.272   9.215  43.201 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) -17.5791     6.7584  -2.601   0.0123 *  
## cars$speed    3.9324     0.4155   9.464 1.49e-12 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 15.38 on 48 degrees of freedom
## Multiple R-squared:  0.6511, Adjusted R-squared:  0.6438 
## F-statistic: 89.57 on 1 and 48 DF,  p-value: 1.49e-12

Brief Explanation

• head(cars): The first 6 rows of the pre-installed data cars.
• str(cars): The data structure of the pre-installed data cars.
• summary(cars): The summary of the pre-installed data cars.
• plot(cars): A scatter plot of the pre-installed data cars.
  • plot(cars$dist~cars$speed)
  • cars$dist, cars$[[2]], cars[,2] are same
• abline(lm(cars$dist~cars$speed)): Add a regression line of a linear model
• lm(cars$dist~cars$speed): The equation of the regression line
• summary(lm(cars$dist~cars$speed): The summary of the linear regression model

hist(cars$dist)

hist(cars$speed)

View and help

• View(cars)
• ?cars: same as help(cars)
• ??cars: same as `help.search(“cars”)

datasets

• ?datasets

• library(help = "datasets")

• data() shows all data already attached and available.

Practicum

Pick a data in the datasets package and try

• head()
• str()
• summary()

and some more.

iris

head(iris)

str(iris)

## 'data.frame':    150 obs. of  5 variables:
##  $ Sepal.Length: num  5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...
##  $ Sepal.Width : num  3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...
##  $ Petal.Length: num  1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...
##  $ Petal.Width : num  0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...
##  $ Species     : Factor w/ 3 levels "setosa","versicolor",..: 1 1 1 1 1 1 1 1 1 1 ...

summary(iris)

##   Sepal.Length    Sepal.Width     Petal.Length    Petal.Width   
##  Min.   :4.300   Min.   :2.000   Min.   :1.000   Min.   :0.100  
##  1st Qu.:5.100   1st Qu.:2.800   1st Qu.:1.600   1st Qu.:0.300  
##  Median :5.800   Median :3.000   Median :4.350   Median :1.300  
##  Mean   :5.843   Mean   :3.057   Mean   :3.758   Mean   :1.199  
##  3rd Qu.:6.400   3rd Qu.:3.300   3rd Qu.:5.100   3rd Qu.:1.800  
##  Max.   :7.900   Max.   :4.400   Max.   :6.900   Max.   :2.500  
##        Species  
##  setosa    :50  
##  versicolor:50  
##  virginica :50  
##                 
##                 
## 

Can you plot?

plot(iris$Sepal.Length, iris$Sepal.Width)

Brief Introduction to R on RStudio

Four Panes and Tabs

1. Top Left: Source Editor
2. Top Right: Environment, History, etc.
3. Bottom Left: Console, Terminal, Render, Background Jobs
4. Bottom Right: Files, Plots, Packages, Help, Viewer, Presentation

Set up

• Highly recommend to set the language to be “English”.
• Create “data” directory.

Sys.setenv(LANG = "en")
dir.create("data")

Three Ways to Run Codes

1. Console - Bottom Left Pane
2. R Script - pull down menu under File
3. R Notebook, R Markdown - pull down menu under File

Second Way: R Script

Examples: R Scripts in Moodle

• basics.R
• coronavirus.R

1. Copy a script in Moodle: {file name}.R
2. In RStudio (create Project in RStudio) choose File > New File > R Script and paste it.
3. Choose File > Save with a name; e.g. {file names} (.R will be added automatically)

To run a code: at the cursor press Ctrl+Shift+Enter (Win) or Cmd+Shift+Enter (Mac).

Packages

R packages are extensions to the R statistical programming language. R packages contain code, data, and documentation in a standardised collection format that can be installed by users of R, typically via a centralised software repository such as CRAN (the Comprehensive R Archive Network).

Installation and attachement

You can install packages by “Install Packages…” under “Tool” in the top menu.

• install.packages("tidyverse")
• install.packages("rmarkdown")

Third Way: R Notebook

Choose R Notebook from the pull down File menu in the top bar.

Edit YAML

Default* is as follows

---
title: "R Notebook"
output: html_notebook
---

Template

---
title: "Title of R Notebook"
author: "ID and Your Name"
date: "2023-01-09" 
output: 
  html_notebook:
#    number_sections: yes
#    toc: true
#    toc_float: true
---

• Don’t change the format. Indention matters.
• The statement after # is ignored.
• Date is automatically inserted when you compile the file.
• You can replace “2023-01-09” by “2022-12-14” or in any date format surrounded by double quotation marks.
• Section numbers: - default is number_sections: no.
• Table of contents, toc: true - default is toc: false.
• Floating table of contents in HTML output, toc_float: true - default is toc_float: false

Create a Code Chunk to Attach Packages

Insert Chunk in Code pull down menu in the top bar, or use the C button on top. You can use shortcut keys listed under Tools in the top bar.

library(tidyverse)

## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.2 ──
## ✔ ggplot2 3.4.0      ✔ purrr   1.0.0 
## ✔ tibble  3.1.8      ✔ dplyr   1.0.10
## ✔ tidyr   1.2.1      ✔ stringr 1.5.0 
## ✔ readr   2.1.3      ✔ forcats 0.5.2 
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()

Importing data

Let us assign the iris data in the pre-installed package datasets to df_iris. You can give any name starting from an alphabet, though there are some rules.

df_iris <- datasets::iris
class(df_iris)

## [1] "data.frame"

The class of data iris is data.frame, the basic data class of R. You can assign the same data as a tibble, the data class of tidyverse as follows.

tbl_iris <- as_tibble(datasets::iris)
class(tbl_iris)

## [1] "tbl_df"     "tbl"        "data.frame"

• df_iris <- iris can replace df_iris <- datasets::iris because the package datasets is installed and attached as default. Since you may have other data called iris included in a different package or you may have changed iris before, it is safer to specify the name of the package with the name of the data.
• Within R Notebook or in Console, you may get different output, and tf_iris and tbl_iris behave differently. It is because of the default settings of R Markdown.

Look at the data

Several ways to view the data.

The View command open up a window to show the contents of the data and you can use the filter as well.

View(df_iris)

The following simple command also shows the data.

df_iris

The output within R Notebook is a tibble style. Try the same command in Console.

slice(df_iris, 1:10)

1:10

##  [1]  1  2  3  4  5  6  7  8  9 10

Data Structure

Let us look at the structure of the data. You can try str(df_iris) on Console or by adding a code chunk in R Notebook introducing later.

glimpse(df_iris)

## Rows: 150
## Columns: 5
## $ Sepal.Length <dbl> 5.1, 4.9, 4.7, 4.6, 5.0, 5.4, 4.6, 5.0, 4.4, 4.9, 5.4, 4.…
## $ Sepal.Width  <dbl> 3.5, 3.0, 3.2, 3.1, 3.6, 3.9, 3.4, 3.4, 2.9, 3.1, 3.7, 3.…
## $ Petal.Length <dbl> 1.4, 1.4, 1.3, 1.5, 1.4, 1.7, 1.4, 1.5, 1.4, 1.5, 1.5, 1.…
## $ Petal.Width  <dbl> 0.2, 0.2, 0.2, 0.2, 0.2, 0.4, 0.3, 0.2, 0.2, 0.1, 0.2, 0.…
## $ Species      <fct> setosa, setosa, setosa, setosa, setosa, setosa, setosa, s…

There are six types of data in R; Double, Integer, Character, Logical, Raw, Complex.

The names after $ are column names. If you call df_iris$Species, you have the Species column. Species is in the 5th collumn, typeof(df_iris[[5]]) does the same as the next.

df_iris[2,4] =0.2 is the fourth entry of Sepal.Width.

typeof(df_iris$Species)

## [1] "integer"

class(df_iris$Species)

## [1] "factor"

For factors = fct see the R Document or an explanation in Factor in R: Categorical Variable & Continuous Variables.

typeof(df_iris$Sepal.Length)

## [1] "double"

class(df_iris$Sepal.Length)

## [1] "numeric"

Q1. What are the differences ofdf_iris, slice(df_iris, 1:10) and glimpse(df_iris) above?

Q2. What are the differences ofdf_iris, slice(df_iris, 1:10) and glimpse(df_iris) in the console?

Summary of the Data

The following is very convenient to get the summary information of a data.

summary(df_iris)

##   Sepal.Length    Sepal.Width     Petal.Length    Petal.Width   
##  Min.   :4.300   Min.   :2.000   Min.   :1.000   Min.   :0.100  
##  1st Qu.:5.100   1st Qu.:2.800   1st Qu.:1.600   1st Qu.:0.300  
##  Median :5.800   Median :3.000   Median :4.350   Median :1.300  
##  Mean   :5.843   Mean   :3.057   Mean   :3.758   Mean   :1.199  
##  3rd Qu.:6.400   3rd Qu.:3.300   3rd Qu.:5.100   3rd Qu.:1.800  
##  Max.   :7.900   Max.   :4.400   Max.   :6.900   Max.   :2.500  
##        Species  
##  setosa    :50  
##  versicolor:50  
##  virginica :50  
##                 
##                 
## 

Minimum, 1st Quadrant (25%), Median, Mean, 3rd Quadrant (75%), Maximum, and the count of each factor.

Visualizing Data

Scatter Plot

We use ggplot to draw graphs. The scatter plot is a projection of data with two variables \(x\) and \(y\).

ggplot(data = <data>, aes(x = <column name for x>, y = <column name for y>)) +
  geom_point()

ggplot(data = df_iris, aes(x = Sepal.Length, y = Sepal.Width)) +
  geom_point()

ggplot(data = df_iris, aes(x = Sepal.Length, y = Sepal.Width)) +
  geom_point()

Scatter Plot with Labels

Add title and labels adding labs().

ggplot(data = <data>, aes(x = <column name for x>, y = <column name for y>)) +
  geom_point() +
  labs(title = "Title", x = "Label for x", y = "Label for y")

ggplot(data = df_iris, aes(x = Sepal.Length, y = Sepal.Width)) +
  geom_point() + 
  labs(title = "Scatter Plot of Sepal Data of Iris", x = "Sepal Length", y = "Sepal Width")

Scatter Plot with Colors

Add different colors automatically to each species. Can you see each group?

ggplot(data = df_iris, aes(x = Sepal.Length, y = Sepal.Width, color = Species)) +
  geom_point()

Scatter Plot with Shapes

ggplot(data = df_iris, aes(x = Sepal.Length, y = Sepal.Width, shape = Species)) +
  geom_point()

Boxplot

The boxplot compactly displays the distribution of a continuous variable.

ggplot(data = df_iris, aes(x = Species, y = Sepal.Length)) +
  geom_boxplot()

Histogram

Visualize the distribution of a single continuous variable by dividing the x axis into bins and counting the number of observations in each bin. Histograms (geom_histogram()) display the counts with bars.

ggplot(data = df_iris, aes(x = Sepal.Length)) +
  geom_histogram()

Change the number of bins by bins = <number>.

ggplot(data = df_iris, aes(x = Sepal.Length)) +
  geom_histogram(bins = 10)

Data Modeling

Professor Kaizoji will cover the mathematical models and hypothesis testings.

ggplot(data = df_iris, aes(x = Sepal.Length, y = Sepal.Width)) +
  geom_point() +
  geom_smooth(method = "lm", se = FALSE)

Helpful Resources

• Cheat Sheet in RStudio: https://www.rstudio.com/resources/cheatsheets/

  • RStudio IED
  • Base R Cheat Sheet

• ‘Quick R’ by DataCamp: https://www.statmethods.net/management

• An Introduction to R

Assignments - See Moodle

1. Assignment Week 2-1: Introduction Plus Forum
   

• Due: Tuesday, 20 December 2022, 11:59 PM

1. Assignment Week 2-2: Quiz 1 on R Basics

• Due: Tuesday, 20 December 2022, 11:59 PM

• {swirl} website: https://swirlstats.com
• JHU Data Science in coursera uses swirl for exercises.

Swirl Courses

1. R Programming: The basics of programming in R
2. Regression Models: The basics of regression modeling in R
3. Statistical Inference: The basics of statistical inference in R
4. Exploratory Data Analysis: The basics of exploring data in R

You can install other swirl courses as well

• Swirl Courses Organized by Title
• Swirl Courses Organized by Author’s Name
• Github: swirl courses
  • install_course("Course Name Here")

Install and Start Swirl Courses

Three Steps to Start Swirl

install.packages("swirl") # Only the first time.
library(swirl) # Everytime you start swirl
swirl() # Everytime you start or resume swirl

R Programming: The basics of programming in R

 1: Basic Building Blocks      2: Workspace and Files     3: Sequences of Numbers    
 4: Vectors                    5: Missing Values          6: Subsetting Vectors      
 7: Matrices and Data Frames   8: Logic                   9: Functions               
10: lapply and sapply         11: vapply and tapply      12: Looking at Data         
13: Simulation                14: Dates and Times        15: Base Graphics          

Recommended Sections in Order

1, 3, 4, 5, 6, 7, 12, 15, 14, 8, 9, 10, 11, 13, 2

• Section 2 discusses the directories and file systems of a computer
• Sections 9, 10, 11 are for programming

Controling a swirl Session

• … <– That’s your cue to press Enter to continue

• You can exit swirl and return to the R prompt (>) at any time by pressing the Esc key.

• If you are already at the prompt, type bye() to exit and save your progress. When you exit properly, you’ll see a short message letting you know you’ve done so.

When you are at the R prompt (>):

1. Typing skip() allows you to skip the current question.
2. Typing play() lets you experiment with R on your own; swirl will ignore what you do…
3. UNTIL you type nxt() which will regain swirl’s attention.
4. Typing bye() causes swirl to exit. Your progress will be saved.
5. Typing main() returns you to swirl’s main menu.
6. Typing info() displays these options again.

Final Remark

You will encounter the message like ‘Would you like to receive credit for completing this course on Coursera.org?’ at the end of each course. This is for coursera courses. Select ‘NO’.

R Scripts in Moodle

• basics.R
• coronavirus.R

1. Copy a script in Moodle: {file name}.R
2. In RStudio (Workspace in RStudio.cloud, Project in RStudio) choose File > New File > R Script and paste it.
3. Choose File > Save with a name; e.g. {file names} (.R will be added automatically)

basics.R

The script with the outputs.

#################
#
# basics.R
#
################
# 'Quick R' by DataCamp may be a handy reference: 
#     https://www.statmethods.net/management/index.html
# Cheat Sheet at RStudio: https://www.rstudio.com/resources/cheatsheets/
# Base R Cheat Sheet: https://github.com/rstudio/cheatsheets/raw/main/base-r.pdf
# To execute the line: Control + Enter (Window and Linux), Command + Enter (Mac)
## try your experiments on the console

## calculator

3 + 7

### +, -, *, /, ^ (or **), %%, %/%

3 + 10 / 2

3^2

2^3

2*2*2

### assignment: <-, (=, ->, assign()) 

x <- 5

x 

#### object_name <- value, '<-' shortcut: Alt (option) + '-' (hyphen or minus) 
#### Object names must start with a letter and can only contain letter, numbers, _ and .

this_is_a_long_name <- 5^3

this_is_a_long_name

char_name <- "What is your name?"

char_name

#### Use 'tab completion' and 'up arrow'

### ls(): list of all assignments

ls()
ls.str()

#### check Environment in the upper right pane

### (atomic) vectors

5:10

a <- seq(5,10)

a

b <- 5:10

identical(a,b)

seq(5,10,2) # same as seq(from = 5, to = 10, by = 2)

c1 <- seq(0,100, by = 10)

c2 <- seq(0,100, length.out = 10)

c1

c2

length(c1)

#### ? seq   ? length   ? identical

(die <- 1:6)

zero_one <- c(0,1) # same as 0:1

die + zero_one # c(1,2,3,4,5,6) + c(0,1). re-use

d1 <- rep(1:3,2) # repeat


d1

die == d1

d2 <- as.character(die == d1)

d2

d3 <- as.numeric(die == d1)

d3

### class() for class and typeof() for mode
### class of vectors: numeric, charcters, logical
### types of vectors: doubles, integers, characters, logicals (complex and raw)

typeof(d1); class(d1)

typeof(d2); class(d2)

typeof(d3); class(d3)

sqrt(2)

sqrt(2)^2

sqrt(2)^2 - 2

typeof(sqrt(2))

typeof(2)

typeof(2L)

5 == c(5)

length(5)

### Subsetting

(A_Z <- LETTERS)

A_F <- A_Z[1:6]

A_F

A_F[3]

A_F[c(3,5)]

large <- die > 3

large

even <- die %in% c(2,4,6)

even

A_F[large]

A_F[even]

A_F[die < 4]

### Compare df with df1 <- data.frame(number = die, alphabet = A_F)
df <- data.frame(number = die, alphabet = A_F, stringsAsFactors = FALSE)

df

df$number

df$alphabet

df[3,2]

df[4,1]

df[1]

class(df[1])

class(df[[1]])

identical(df[[1]], die)

identical(df[1],die)

####################
# The First Example
####################

plot(cars)

# Help

? cars

# cars is in the 'datasets' package

data()

# help(cars) does the same as ? cars
# You can use Help tab in the right bottom pane

help(plot)
? par

head(cars)

str(cars)

summary(cars)

x <- cars$speed
y <- cars$dist

min(x)
mean(x)
quantile(x)

plot(cars)

abline(lm(cars$dist ~ cars$speed))

summary(lm(cars$dist ~ cars$speed))

boxplot(cars)

hist(cars$speed)
hist(cars$dist)
hist(cars$dist, breaks = seq(0,120, 10))

coronavirus.R

The script and its outputs. coronavirus.csv is very large

# https://coronavirus.jhu.edu/map.html
# JHU Covid-19 global time series data
# See R pakage coronavirus at: https://github.com/RamiKrispin/coronavirus
# Data taken from: https://github.com/RamiKrispin/coronavirus/tree/master/csv
# Last Updated
Sys.Date()

## Download and read csv (comma separated value) file
coronavirus <- read.csv("https://github.com/RamiKrispin/coronavirus/raw/master/csv/coronavirus.csv")
# write.csv(coronavirus, "data/coronavirus.csv")

## Summaries and structures of the data
head(coronavirus)
str(coronavirus)
coronavirus$date <- as.Date(coronavirus$date)
str(coronavirus)

range(coronavirus$date)
unique(coronavirus$country)
unique(coronavirus$type)

## Set Country
COUNTRY <- "Japan"
df0 <- coronavirus[coronavirus$country == COUNTRY,]
head(df0)
tail(df0)
(pop <- df0$population[1])
df <- df0[c(1,6,7,13)]
str(df)
head(df)
### alternatively,
head(df0[c("date", "type", "cases", "population")])
###

## Set types
df_confirmed <- df[df$type == "confirmed",]
df_death <- df[df$type == "death",]
df_recovery <- df[df$data_type == "recovery",]
head(df_confirmed)
head(df_death)
head(df_recovery)

## Histogram
plot(df_confirmed$date, df_confirmed$cases, type = "h")
plot(df_death$date, df_death$cases, type = "h")
# plot(df_recovered$date, df_recovered$cases, type = "h") # no data for recovery

## Scatter plot and correlation
plot(df_confirmed$cases, df_death$cases, type = "p")
cor(df_confirmed$cases, df_death$cases)


## In addition set a period
start_date <- as.Date("2021-07-01")
end_date <- Sys.Date() 
df_date <- df[df$date >=start_date & df$date <= end_date,]
##

## Set types
df_date_confirmed <- df_date[df_date$type == "confirmed",]
df_date_death <- df_date[df_date$type == "death",]
df_date_recovery <- df_date[df_date$data_type == "recovery",]
head(df_date_confirmed)
head(df_date_death)
head(df_date_recovery)

## Histogram
plot(df_date_confirmed$date, df_date_confirmed$cases, type = "h")
plot(df_date_death$date, df_date_death$cases, type = "h")
# plot(df_date_recovered$date, df_date_recovered$cases, type = "h") # no data for recovery

plot(df_date_confirmed$cases, df_date_death$cases, type = "p")
cor(df_date_confirmed$cases, df_date_death$cases)

### Q0. Change the values of the location and the period and see the outcomes.
### Q1. What is the correlation between df_confirmed$cases and df_death$cases?
### Q2. Do we have a larger correlation value if we shift the dates to implement the time-lag?
### Q3. Do you have any other questions to explore?

#### Extra
plot(df_confirmed$date, df_confirmed$cases, type = "h", 
     main = paste("Comfirmed Cases in",COUNTRY), 
     xlab = "Date", ylab = "Number of Cases")

:::

Hans Rosling (1948 – 2017)

Hans Rosling was a Swedish physician, academic, and public speaker. He was a professor of international health at Karolinska Institute[4] and was the co-founder and chairman of the Gapminder Foundation, which developed the Trendalyzer software system. (wikipedia)

• Books:
  • Factfulness: Ten Reasons We’re Wrong About The World - And Why Things Are Better Than You Think, 2018
  • How I Learned to Understand the World: A Memoir, 2020
• Gapminder: https://www.gapminder.org
  • You are probably wrong about: Upgrade Your World View
  • Bubble Chart: Income vs Life Expectancy over time, 1800 - 2020
    • How many variables?
• Videos: The best stats you’ve ever seen, Hans Rosling

Factfulness is … From the book

recognizing when a decision feels urgent and remembering that it rarely is.

To control the urgency instinct, take small steps.

• Take a breath. When your urgency instinct is triggered, your other instincts kick in and your analysis shuts down. Ask for more time and more information. It’s rarely now or never and it’s rarely either/or.
• Insist on the data. If something is urgent and important, it should be measured. Beware of data that is relevant but inaccurate, or accurate but irrelevant. Only relevant and accurate data is useful.
• Beware of fortune-tellers. Any prediction about the future is uncertain. Be wary of predictions that fail to acknowledge that. Insist on a full range of scenarios, never just the best or worst case. Ask how often such predictions have been right before.
• Be wary of drastic action. Ask what the side effects will be. Ask how the idea has been tested. Step-by-step practical improvements, and evaluation of their impact, are less dramatic but usually more effective.

# install.packages("gapminder")
library(gapminder)

df <- gapminder
df

glimpse(df)

## Rows: 1,704
## Columns: 6
## $ country   <fct> "Afghanistan", "Afghanistan", "Afghanistan", "Afghanistan", …
## $ continent <fct> Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, …
## $ year      <int> 1952, 1957, 1962, 1967, 1972, 1977, 1982, 1987, 1992, 1997, …
## $ lifeExp   <dbl> 28.801, 30.332, 31.997, 34.020, 36.088, 38.438, 39.854, 40.8…
## $ pop       <int> 8425333, 9240934, 10267083, 11537966, 13079460, 14880372, 12…
## $ gdpPercap <dbl> 779.4453, 820.8530, 853.1007, 836.1971, 739.9811, 786.1134, …

summary(df)

##         country        continent        year         lifeExp     
##  Afghanistan:  12   Africa  :624   Min.   :1952   Min.   :23.60  
##  Albania    :  12   Americas:300   1st Qu.:1966   1st Qu.:48.20  
##  Algeria    :  12   Asia    :396   Median :1980   Median :60.71  
##  Angola     :  12   Europe  :360   Mean   :1980   Mean   :59.47  
##  Argentina  :  12   Oceania : 24   3rd Qu.:1993   3rd Qu.:70.85  
##  Australia  :  12                  Max.   :2007   Max.   :82.60  
##  (Other)    :1632                                                
##       pop              gdpPercap       
##  Min.   :6.001e+04   Min.   :   241.2  
##  1st Qu.:2.794e+06   1st Qu.:  1202.1  
##  Median :7.024e+06   Median :  3531.8  
##  Mean   :2.960e+07   Mean   :  7215.3  
##  3rd Qu.:1.959e+07   3rd Qu.:  9325.5  
##  Max.   :1.319e+09   Max.   :113523.1  
## 

Questions

• List questions based on this data.
• What do you want to see?
• What kind of chart do you want to construct?

EDA (A diagram from R4DS by H.W. and G.G.)

EDA from r4ds

Today: R Markdown and dplyr

What is R Markdown: https://vimeo.com/178485416

• Code Chunks
• Text
• YAML Metadata

What is R Markdown and R Notebook

R Markdown provides an authoring framework for data science. You can use a single R Markdown file to both

• save and execute code
• generate high quality reports that can be shared with an audience

R Notebooks are an implementation of Literate Programming that allows for direct interaction with R while producing a reproducible document with publication-quality output.

An R Notebook is an R Markdown document with chunks that can be executed independently and interactively, with output visible immediately beneath the input.

(Reference: R Markdown: The Definitive Guide, 3.2 Notebook)

Two Goodies

• Important: Implementation of Reproducible Research and Literate Programming

• Useful to Render into Various Formats: R Notebook (HTML), R Markdown (HTML), PDF, MS Word, MS Powerpoint, Ioslides Presentation (HTML), Slidy Presentation (HTML), Beamer Presentation (PDF), etc.

Reproducible Research and Literate Programming

Literate Programming by D. Knuth

Literate programming is an approach to programming introduced by Donald Knuth in which a program is given as an explanation of the program logic in a natural language, such as English, interspersed with snippets of macros and traditional source code, from which a compilable source code can be generated

D. Knuth

Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.

Reproducible Research - Quote from a Coursera Course

Reproducible research is the idea that data analyses, and more generally, scientific claims, are published with their data and software code so that others may verify the findings and build upon them. The need for reproducibility is increasing dramatically as data analyses become more complex, involving larger datasets and more sophisticated computations. Reproducibility allows for people to focus on the actual content of a data analysis, rather than on superficial details reported in a written summary. In addition, reproducibility makes an analysis more useful to others because the data and code that actually conducted the analysis are available.

R Markdown workflow, R for Data Science

R Markdown is also important because it so tightly integrates prose and code. This makes it a great analysis notebook because it lets you develop code and record your thoughts. It:

• Records what you did and why you did it. Regardless of how great your memory is, if you don’t record what you do, there will come a time when you have forgotten important details. Write them down so you don’t forget!

• Supports rigorous thinking. You are more likely to come up with a strong analysis if you record your thoughts as you go, and continue to reflect on them. This also saves you time when you eventually write up your analysis to share with others.

• Helps others understand your work. It is rare to do data analysis by yourself, and you’ll often be working as part of a team. A lab notebook helps you share why you did it with your colleagues or lab mates.

Records of EDA and Communication

1. Memo on a scratch paper: R Scripts
2. Record on a notebook: R Notebook (an R Markdown format)
3. Short paper or a digital communication: R Notebook
4. Paper or a report: R Markdown (html, pdf, MS Word, etc.)
5. Presentation (html, pdf, MS Powerpoint, etc.)
6. Publication of a Book

• BOOKDOWN: Write HTML, PDF, ePub, and Kindle books with R Markdown. Free online document is provided in pdf as well
• Arxive Page

Let’s Get Started

1. Start R Studio - Update R Studio if old
2. Create a Project
3. Tool > Install Packages rmarkdown
   • Or on Console: install.packages("rmarkdown")
4. Tool > Install Packages tinytex (for pdf generation)
   • Alternatively, install.packages('tinytex')
   • If TeX is not installed: tinytex::install_tinytex() # install TinyTeX
     • If you are not sure, please check on Terminal in the left below pane:
       • which latex, which mktexlsr
5. Let’s try!
   1. File > New File > R Notebook
   2. Save with a file name, say, test-notebook
   3. Preview by [Preview] button
   4. Run Code Chunk plot(cars) and then Preview again.
   5. Knit PDF, Word (and HTML)

Templates

RNotebook_Template

Template to submit your assignment of this course: RNotebook_Template.nb.html

title: "Title of R Notebook"
author: "ID and Your Name"
date: "2023-01-09" 
output:
  html_notebook: null

YAML

• Change the title
• Write ID and your name
• Date is auto-generated and inserted. If you wish, you can replace “2023-01-09” by your favorite date style.

Code Chunk

• When you execute or run a code within the notebook, the results appear beneath the code.
• Try executing this chunk by clicking the Run button, a triangle pointing right, within the chunk or by placing your cursor inside it and pressing Ctrl+Shift+Enter (Win) or Cmd+Shift+Enter (Mac).
• Add a new chunk by clicking the Insert Chunk button on the toolbar or by pressing Ctrl+Option+I (Win) or Cmd+Option+I (Mac).
• When you save the notebook, an HTML file containing the code and output will be saved alongside it (click the Preview button or press Ctrl+Shift+K (Win) or Cmd+Shift+K (Mac) to preview the HTML file).
• The preview shows you a rendered HTML copy of the contents of the editor. Consequently, unlike Knit, Preview does not run any R code chunks. Instead, the output of the chunk when it was last run in the editor is displayed.

Testing R Markdown Formats

Various Output Formats: test-rmarkdown.nb.html

title: "Testing R Markdown Formats"
author: "DS-SL"
date: "2023-01-09"
output:
  html_notebook:
    number_sections: yes
  pdf_document: 
    number_sections: yes
  html_document:
    df_print: paged
    number_sections: yes
  word_document: 
    number_sections: yes
  powerpoint_presentation: default
  ioslides_presentation:
    widescreen: yes
    smaller: yes
  slidy_presentation: default
  beamer_presentation: default

Comments on Presentation Formats and Options

• For slides, a new slide starts at ##, the second-level heading.
• --- is page break for presentation formats.
• For Word and Powerpoint, you can add your template. See the documents in References
  • Use R Markdown to create a Word document [similar for PowerPoint]
  • Save the rendered Word file as: ref-doc-style.docx
  • Edit the styles of the file ref-doc-style.docx
  • Add ref-doc-style.docx as reference_doc in YAML with indention as below

  word_document: 
    number_sections: yes
    reference_doc: ref-doc-style.docx
  powerpoint_presentation: 
    reference_doc: ref-ppt-style.pptx

• You can use Output Options at the bottom of the gear icon next to Preview/knit button.

Markdown Language – or use WYSIWYG editor

• Headers: #, ##, ###, ####
• Lists: 1. 2. , *
• Links: linked phrase
• Images: ![alt text](figures/filename.jpg)
• Block quotes” > (block)
•   equations: e.g. $\frac{a}{b}$ for \(\frac{a}{b}\)
• Horizontal rules: Three or more asterisks or dashes (*** or - - - )
• Tables
• Footnotes
• Bibliographies and Citations
• Slide breaks
• Italicized text by _italic_, Bold text by **bold**
• Superscripts, Subscripts, Strikethrough text

Visual R Markdown

R Studio introduced Visual Editor towards the end of 2021. It seems to be stable but it is not perfect to go back and forth from the original editor using tags. I always use the original editor and I am confident on all the functions of it but I do not have much experience on Visual Editor. [My Note in QALL401 2021]

• https://rstudio.github.io/visual-markdown-editing/

References

• Posit Primers: Report Reproducibly
• Markdown Quick Reference: Top Menu Bar > Help > Markdown Quick Reference
• Cheat Sheet (Top Menu Bar: Help > Cheat Sheets): RMarkdown Cheat Sheet, RMarkdown Reference Guide
• Books:
  • In Textbook: R for Data Science: Communicate
  • R Markdown: The Definitive Guide by Yihui Xie, J. J. Allaire, Garrett Grolemund
  • R Markdown Cookbook by Yihui Xie, Christophe Dervieux, Emily Riederer
• Markdown: R Markdown is based on the Markdown language of Pandoc
  • Pandoc’s Markdown: Detailed Information
  • Markdown Tutorials: Interactive Practicum
  • DARING FIREBALL: Markdown (detailed explanation and editor as Dingus)
• Post error messages to a web search engine.

dplyr Overview

dplyr is a grammar of data manipulation, providing a consistent set of verbs that help you solve the most common data manipulation challenges:

• select() picks variables based on their names.
• filter() picks cases based on their values.
• mutate() adds new variables that are functions of existing variables
• summarise() reduces multiple values down to a single summary.
• arrange() changes the ordering of the rows.
• group_by() takes an existing tbl and converts it into a grouped tbl.

You can learn more about them in vignette(“dplyr”). As well as these single-table verbs, dplyr also provides a variety of two-table verbs, which you can learn about in vignette(“two-table”).

If you are new to dplyr, the best place to start is the data transformation chapter in R for data science.

select: Subset columns using their names and types

filter: Subset rows using column values

arrange and Pipe %>%

• arrange() orders the rows of a data frame by the values of selected columns.

Unlike other dplyr verbs, arrange() largely ignores grouping; you need to explicitly mention grouping variables (`or use .by_group = TRUE) in order to group by them, and functions of variables are evaluated once per data frame, not once per group.

• pipes in R for Data Science.

mutate

• Create, modify, and delete columns

• Useful mutate functions

  • +, -, log(), etc., for their usual mathematical meanings

  • lead(), lag()

  • dense_rank(), min_rank(), percent_rank(), row_number(), cume_dist(), ntile()

  • cumsum(), cummean(), cummin(), cummax(), cumany(), cumall()

  • na_if(), coalesce()### group_by() and summarise()

group_by

summarise or summarize

Summary functions

So far our summarise() examples have relied on sum(), max(), and mean(). But you can use any function in summarise() so long as it meets one criteria: the function must take a vector of values as input and return a single value as output. Functions that do this are known as summary functions and they are common in the field of descriptive statistics. Some of the most useful summary functions include:

1. Measures of location - mean(x), median(x), quantile(x, 0.25), min(x), and max(x)
2. Measures of spread - sd(x), var(x), IQR(x), and mad(x)
3. Measures of position - first(x), nth(x, 2), and last(x)
4. Counts - n_distinct(x) and n(), which takes no arguments, and returns the size of the current group or data frame.
5. Counts and proportions of logical values - sum(!is.na(x)), which counts the number of TRUEs returned by a logical test; mean(y == 0), which returns the proportion of TRUEs returned by a logical test.

• if_else(), recode(), case_when()

Learn dplyr by Examples

Data iris

iris

summary(iris)

##   Sepal.Length    Sepal.Width     Petal.Length    Petal.Width   
##  Min.   :4.300   Min.   :2.000   Min.   :1.000   Min.   :0.100  
##  1st Qu.:5.100   1st Qu.:2.800   1st Qu.:1.600   1st Qu.:0.300  
##  Median :5.800   Median :3.000   Median :4.350   Median :1.300  
##  Mean   :5.843   Mean   :3.057   Mean   :3.758   Mean   :1.199  
##  3rd Qu.:6.400   3rd Qu.:3.300   3rd Qu.:5.100   3rd Qu.:1.800  
##  Max.   :7.900   Max.   :4.400   Max.   :6.900   Max.   :2.500  
##        Species  
##  setosa    :50  
##  versicolor:50  
##  virginica :50  
##                 
##                 
## 

select 1 - columns 1, 2, 5

select(iris, c(1,2,5))

select 2 - except Species

select(iris, -Species)

select 3 - change column names

select(iris, sl = Sepal.Length, sw = Sepal.Width, sp = Species)

filter - by names

filter(iris, Species == "virginica")

arrange - ascending and descending order

arrange(iris, Sepal.Length, desc(Sepal.Width))

mutate - rank

iris %>% mutate(sl_rank = min_rank(Sepal.Length)) %>% arrange(sl_rank)

group_by and summarize

iris %>% 
  group_by(Species) %>% 
  summarize(sl = mean(Sepal.Length), sw = mean(Sepal.Width), 
  pl = mean(Petal.Length), pw = mean(Petal.Width))

• mean: mean() or mean(x, na.rm = TRUE) - arithmetic mean (average)
• median: median() or median(x, na.rm = TRUE) - mid value

For more examples see

dplr_iris

References of dplyr

• Textbook: R for Data Science, Part II Explore

Learn dplyr by Examples II - gapminder

ggplot2 Overview

ggplot2 is a system for declaratively creating graphics, based on The Grammar of Graphics. You provide the data, tell ggplot2 how to map variables to aesthetics, what graphical primitives to use, and it takes care of the details.

Examples

ggplot(data = mpg) + 
  geom_point(mapping = aes(x = displ, y = hwy))

ggplot(data = mpg) + 
  geom_boxplot(mapping = aes(x = class, y = hwy))

Template

ggplot(data = <DATA>) + 
  <GEOM_FUNCTION>(mapping = aes(<MAPPINGS>))

Gapminder and R Package gapminder

Gapminder was founded by Ola Rosling, Anna Rosling Rönnlund, and Hans Rosling

• Gapminder: https://www.gapminder.org

  • Test on Top: You are probably wrong about - upgrade your worldview
  • Bubble Chart: https://www.gapminder.org/tools/#$chart-type=bubbles&url=v1
  • Dallar Street: https://www.gapminder.org/tools/#$chart-type=bubbles&url=v1
  • Data: https://www.gapminder.org/data/

• R Package gapminder by Jennifer Bryan

  • Package site: https://CRAN.R-project.org/package=gapminder
  • Site: https://github.com/jennybc/gapminder
  • Documents: https://www.rdocumentation.org/packages/gapminder/versions/0.3.0

• Package Help ?gapminder or gapminder in the search window of Help

  • The main data frame gapminder has 1704 rows and 6 variables:
    • country: factor with 142 levels
    • continent: factor with 5 levels
    • year: ranges from 1952 to 2007 in increments of 5 years
    • lifeExp: life expectancy at birth, in years
    • pop: population
    • gdpPercap: GDP per capita (US$, inflation-adjusted)

library(tidyverse)
library(gapminder)
library(WDI)

R Package gapminder data

df <- gapminder
df

glimpse(df)

## Rows: 1,704
## Columns: 6
## $ country   <fct> "Afghanistan", "Afghanistan", "Afghanistan", "Afghanistan", …
## $ continent <fct> Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, …
## $ year      <int> 1952, 1957, 1962, 1967, 1972, 1977, 1982, 1987, 1992, 1997, …
## $ lifeExp   <dbl> 28.801, 30.332, 31.997, 34.020, 36.088, 38.438, 39.854, 40.8…
## $ pop       <int> 8425333, 9240934, 10267083, 11537966, 13079460, 14880372, 12…
## $ gdpPercap <dbl> 779.4453, 820.8530, 853.1007, 836.1971, 739.9811, 786.1134, …

summary(df)

##         country        continent        year         lifeExp     
##  Afghanistan:  12   Africa  :624   Min.   :1952   Min.   :23.60  
##  Albania    :  12   Americas:300   1st Qu.:1966   1st Qu.:48.20  
##  Algeria    :  12   Asia    :396   Median :1980   Median :60.71  
##  Angola     :  12   Europe  :360   Mean   :1980   Mean   :59.47  
##  Argentina  :  12   Oceania : 24   3rd Qu.:1993   3rd Qu.:70.85  
##  Australia  :  12                  Max.   :2007   Max.   :82.60  
##  (Other)    :1632                                                
##       pop              gdpPercap       
##  Min.   :6.001e+04   Min.   :   241.2  
##  1st Qu.:2.794e+06   1st Qu.:  1202.1  
##  Median :7.024e+06   Median :  3531.8  
##  Mean   :2.960e+07   Mean   :  7215.3  
##  3rd Qu.:1.959e+07   3rd Qu.:  9325.5  
##  Max.   :1.319e+09   Max.   :113523.1  
## 

Tidyverse::ggplot

First Try - with failures

ggplot(df, aes(x = year, y = lifeExp)) + geom_point()

ggplot(df, aes(x = year, y = lifeExp)) + geom_line()

ggplot(df, aes(x = year, y = lifeExp)) + geom_boxplot()

## Warning: Continuous x aesthetic
## ℹ did you forget `aes(group = ...)`?

typeof(pull(df, year)) # same as typeof(df$year)

## [1] "integer"

ggplot(df, aes(y = lifeExp, group = year)) + geom_boxplot()

Box Plot

ggplot(df, aes(x = as_factor(year), y = lifeExp)) + geom_boxplot()

Applications of dplyr

filter

df %>% filter(country == "Afghanistan") %>%
  ggplot(aes(x = year, y = lifeExp)) + geom_line()

df %>% filter(country %in% c("Afghanistan", "Japan")) %>%
  ggplot(aes(x = year, y = lifeExp, color = country)) + geom_line()

df %>% distinct(country) %>% pull()

##   [1] Afghanistan              Albania                  Algeria                 
##   [4] Angola                   Argentina                Australia               
##   [7] Austria                  Bahrain                  Bangladesh              
##  [10] Belgium                  Benin                    Bolivia                 
##  [13] Bosnia and Herzegovina   Botswana                 Brazil                  
##  [16] Bulgaria                 Burkina Faso             Burundi                 
##  [19] Cambodia                 Cameroon                 Canada                  
##  [22] Central African Republic Chad                     Chile                   
##  [25] China                    Colombia                 Comoros                 
##  [28] Congo, Dem. Rep.         Congo, Rep.              Costa Rica              
##  [31] Cote d'Ivoire            Croatia                  Cuba                    
##  [34] Czech Republic           Denmark                  Djibouti                
##  [37] Dominican Republic       Ecuador                  Egypt                   
##  [40] El Salvador              Equatorial Guinea        Eritrea                 
##  [43] Ethiopia                 Finland                  France                  
##  [46] Gabon                    Gambia                   Germany                 
##  [49] Ghana                    Greece                   Guatemala               
##  [52] Guinea                   Guinea-Bissau            Haiti                   
##  [55] Honduras                 Hong Kong, China         Hungary                 
##  [58] Iceland                  India                    Indonesia               
##  [61] Iran                     Iraq                     Ireland                 
##  [64] Israel                   Italy                    Jamaica                 
##  [67] Japan                    Jordan                   Kenya                   
##  [70] Korea, Dem. Rep.         Korea, Rep.              Kuwait                  
##  [73] Lebanon                  Lesotho                  Liberia                 
##  [76] Libya                    Madagascar               Malawi                  
##  [79] Malaysia                 Mali                     Mauritania              
##  [82] Mauritius                Mexico                   Mongolia                
##  [85] Montenegro               Morocco                  Mozambique              
##  [88] Myanmar                  Namibia                  Nepal                   
##  [91] Netherlands              New Zealand              Nicaragua               
##  [94] Niger                    Nigeria                  Norway                  
##  [97] Oman                     Pakistan                 Panama                  
## [100] Paraguay                 Peru                     Philippines             
## [103] Poland                   Portugal                 Puerto Rico             
## [106] Reunion                  Romania                  Rwanda                  
## [109] Sao Tome and Principe    Saudi Arabia             Senegal                 
## [112] Serbia                   Sierra Leone             Singapore               
## [115] Slovak Republic          Slovenia                 Somalia                 
## [118] South Africa             Spain                    Sri Lanka               
## [121] Sudan                    Swaziland                Sweden                  
## [124] Switzerland              Syria                    Taiwan                  
## [127] Tanzania                 Thailand                 Togo                    
## [130] Trinidad and Tobago      Tunisia                  Turkey                  
## [133] Uganda                   United Kingdom           United States           
## [136] Uruguay                  Venezuela                Vietnam                 
## [139] West Bank and Gaza       Yemen, Rep.              Zambia                  
## [142] Zimbabwe                
## 142 Levels: Afghanistan Albania Algeria Angola Argentina Australia ... Zimbabwe

df %>% filter(country %in% c("Brazil", "Russia", "India", "China")) %>%
  ggplot(aes(x = year, y = lifeExp, color = country)) + geom_line()

Russian data is missing.

Exercises

1. Change lifeExp to pop and gdpPercap and do the same.
2. Choose ASEAN countries and do the similar investigations.

• Brunei, Cambodia, Indonesia, Laos, Malaysia, Myanmar, Philippines, Singapore.

1. Choose several countries by yourself and do the similar investigations.

group_by and summarize

Let us use the variable continent and summarize the data.

df_lifeExp <- df %>% group_by(continent, year) %>% 
  summarize(mean_lifeExp = mean(lifeExp), median_lifeExp = median(lifeExp), max_lifeExp = max(lifeExp), min_lifeExp = min(lifeExp), .groups = "keep")

df_lifeExp

df %>% filter(year %in% c(1952, 1987, 2007)) %>%
  ggplot(aes(x=as_factor(year), y = lifeExp, fill = continent)) +
  geom_boxplot()

df_lifeExp %>% ggplot(aes(x = year, y = mean_lifeExp, color = continent)) +
  geom_line()

df_lifeExp %>% ggplot(aes(x = year, y = mean_lifeExp, color = continent, linetype = continent)) +
  geom_line()

df_lifeExp %>% ggplot() +
  geom_line(aes(x = year, y = mean_lifeExp, color = continent)) + 
  geom_line(aes(x = year, y = median_lifeExp, linetype = continent))

R Markdown and dplyr

• Create an R Notebook of a Data Analysis containing the following and submit the rendered HTML file (eg. a2_123456.nb.html)
  1. create an R Notebook using the R Notebook Template in Moodle, save as a2_123456.Rmd,
  2. write your name and ID and the contents,
  3. run each code block,
  4. preview to create a2_123456.nb.html,
  5. submit a2_123456.nb.html to Moodle.

1. Pick data from the built-in datasets besides cars. (library(help = "datasets") or go to the site The R Datasets Package)

   • Information of the data: Name, Description, Usage, Format, Source, References (Hint: ?cars)
   • Use head(), str(), …, and create at least one chart using ggplot2 - Code Chunk.
     • Don’t forget to add library(tidyverse) in the first code chunk.
   • An observation of the chart - in your own words.

1. Load gapminder by library(gapminder).

   • Choose pop or gdpPercap, or both, one country in the data, a group of countries in the data.
   • Create charts using ggplot2 with geom_line and the variables and countries chosen in 1. (See examples of the charts for lifeExp.)
   • Study the data as you like.
   • Observations and difficulties encountered.

Due: 2023-01-09 23:59:00. Submit your R Notebook file in Moodle (The Second Assignment). Due on Monday!

Original Data? WDI?

gapminder

WDI

• SP.DYN.LE00.IN: Life expectancy at birth, total (years)
• NY.GDP.PCAP.KD: GDP per capita (constant 2015 US$)
• SP.POP.TOTL: Population, total

df_wdi <- WDI(
  country = "all", 
  indicator = c(lifeExp = "SP.DYN.LE00.IN", pop = "SP.POP.TOTL", gdpPercap = "NY.GDP.PCAP.KD")
)

df_wdi

df_wdi_extra <- WDI(
  country = "all", 
  indicator = c(lifeExp = "SP.DYN.LE00.IN", pop = "SP.POP.TOTL", gdpPercap = "NY.GDP.PCAP.KD"), 
  extra = TRUE
)

df_wdi_extra

Reviews and Previews

library(tidyverse)
library(gapminder)
library(maps)
library(WDI)

(df <- gapminder)

asean <- c("Brunei", "Cambodia", "Laos", "Myanmar", "Philippines", "Indonesia", "Malaysia", "Singapore")
df %>% filter(country %in% asean) %>%
  ggplot(aes(x = year, y = gdpPercap, col = country)) + geom_line()

df %>% filter(country %in% asean) %>%
  ggplot(aes(x = gdpPercap, y = lifeExp, col = country)) + geom_point()

df %>% filter(country %in% asean) %>%
  ggplot(aes(x = gdpPercap, y = lifeExp, col = country)) + 
  geom_point() + coord_trans(x = "log10", y = "identity")

\(\log_{10}{100}\) = 2, \(\log_{10}{1000}\) = 3, \(\log_{10}{10000}\) = 4

\(10^{2.5}\) = 316.227766, \(10^{3}\) = 1000, \(10^{3.5}\) = 3162.2776602,

\(10^{4}\) = 10^{4}, \(10^{4.5}\) = 3.1622777^{4}.

WDI

• SP.DYN.LE00.IN: Life expectancy at birth, total (years)
• NY.GDP.PCAP.KD: GDP per capita (constant 2015 US$)
• SP.POP.TOTL: Population, total

df_wdi <- WDI(
  country = "all", 
  indicator = c(lifeExp = "SP.DYN.LE00.IN", pop = "SP.POP.TOTL", gdpPercap = "NY.GDP.PCAP.KD")
)

df_wdi

df_wdi_extra <- WDI(
  country = "all", 
  indicator = c(lifeExp = "SP.DYN.LE00.IN", pop = "SP.POP.TOTL", gdpPercap = "NY.GDP.PCAP.KD"), 
  extra = TRUE
)

df_wdi_extra

Exploratory Data Analysis

What is EDA (Posit Primers: Visualise Data)

1. EDA is an iterative cycle that helps you understand what your data says. When you do EDA, you:

2. Generate questions about your data

3. Search for answers by visualising, transforming, and/or modeling your data

Use what you learn to refine your questions and/or generate new questions

EDA is an important part of any data analysis. You can use EDA to make discoveries about the world; or you can use EDA to ensure the quality of your data, asking questions about whether the data meets your standards or not.

Open and Public Data, World Bank

Open Government Data Toolkit: Open Data Defined

The term ``Open Data’’ has a very precise meaning. Data or content is open if anyone is free to use, re-use or redistribute it, subject at most to measures that preserve provenance and openness.

1. The data must be , which means they must be placed in the public domain or under liberal terms of use with minimal restrictions.
2. The data must be , which means they must be published in electronic formats that are machine readable and non-proprietary, so that anyone can access and use the data using common, freely available software tools. Data must also be publicly available and accessible on a public server, without password or firewall restrictions. To make Open Data easier to find, most organizations create and manage Open Data catalogs.

World Bank: WDI - World Development Indicaters

• World Bank: https://www.worldbank.org
• Who we are:
  • To end extreme poverty: By reducing the share of the global population that lives in extreme poverty to 3 percent by 2030.
  • To promote shared prosperity: By increasing the incomes of the poorest 40 percent of people in every country.
• World Bank Open Data: https://data.worldbank.org
• World Development Indicators (WDI) : the World Bank’s premier compilation of cross-country comparable data on development.
  • Poverty and Inequality
  • People
  • Environment
  • Economy
  • States and Markets
  • Global Links

R Package WDI

• WDI: World Development Indicators and Other World Bank Data
• Search and download data from over 40 databases hosted by the World Bank, including the World Development Indicators (‘WDI’), International Debt Statistics, Doing Business, Human Capital Index, and Sub-national Poverty indicators.
• Version: 2.7.4
• Materials: README - usage
  • NEWS - version history
• Published: 2021-04-06
• README: https://cran.r-project.org/web/packages/WDI/readme/README.html
• Reference manual: WDI.pdf

Function WDI

• Usage

WDI(country = "all",
    indicator = "NY.GDP.PCAP.KD",
    start = 1960,
    end = 2020,
    extra = FALSE,
    cache = NULL)

• Arguments
  • country: Vector of countries (ISO-2 character codes, e.g. “BR”, “US”, “CA”, or “all”)
  • indicator: If you supply a named vector, the indicators will be automatically renamed: c('women_private_sector' = 'BI.PWK.PRVS.FE.ZS')

Function WDIsearch

library(WDI)

WDIsearch(string = "NY.GDP.PCAP.KD", 
          field = "indicator", cache = NULL)

WDIsearch(string = "NY.GDP.PCAP.KD", 
  field = "indicator", short = FALSE, cache = NULL)

WDIsearch(string = "gdp", 
  field = "name", short = TRUE, cache = NULL) 

https://databank.worldbank.org/GDP-per-Capita-of-BRICs/id/ec7b40a8

World Development Indicators

readr, readxl

readr, ggplot2; Public Data, WDI, WIR, etc

Learning Resouces

• Posit Primers:
  • Visualize Data: Learn how to use ggplot2 to make any type of plot with your data. Then learn the best ways to visualize patterns within values and relationships between variables.
• r4ds: Explore, II

Exploratory Data Analysis

What is EDA?

EDA is an iterative cycle that helps you understand what your data says. When you do EDA, you:

1. Generate questions about your data

2. Search for answers by visualising, transforming, and/or modeling your data

3. Use what you learn to refine your questions and/or generate new questions

EDA is an important part of any data analysis. You can use EDA to make discoveries about the world; or you can use EDA to ensure the quality of your data, asking questions about whether the data meets your standards or not.

Two useful questions

There is no rule about which questions you should ask to guide your research. However, two types of questions will always be useful for making discoveries within your data. You can loosely word these questions as:

1. What type of variation occurs within my variables?
2. What type of covariation occurs between my variables?

The rest of this tutorial will look at these two questions. To make the discussion easier, let’s define some terms…

Data Visualization

ggplot2 Basics

visualization

World Inequility Report

ggplot2 Extra

Definitions

• A variable is a quantity, quality, or property that you can measure.
• A value is the state of a variable when you measure it. The value of a variable may change from measurement to measurement.
• An observation or case is a set of measurements made under similar conditions (you usually make all of the measurements in an observation at the same time and on the same object). An observation will contain several values, each associated with a different variable. I’ll sometimes refer to an observation as a case or data point.
• Tabular data is a table of values, each associated with a variable and an observation. Tabular data is tidy if each value is placed in its own cell, each variable in its own column, and each observation in its own row.
• So far, all of the data that you’ve seen has been tidy. In real-life, most data isn’t tidy, so we’ll come back to these ideas again in Data Wrangling.

tidyr Basics

WIR2022

tidyr, etc.; WDI, WIR, etc

Other Examples

R Markdown Revisited

Literate Programming and Reproducible Research

Presentation and Paper

1. Data Source

2. Variables

3. Problems

4. Visualization

5. Model

6. Conclusions and Further Research

   WDI, WIR, etc